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A E STB ACT 

The training of psychotherapists has been an ongoing 
process in psychiatry a«d clinical psychology. Becently, however, a 
growing demand to operationalize competence criteria tc enatle 
independent evaluation of therapists,' skills in specif ically defined 
psychotherapies has occurred. To examine thi'S phenomenon, evaluation 
procedures vere developed and tested durin g' a pilot training program, 
in Interpersonal Psychotherapy (IPT). Several different instruments 
and .procedures we're used to monitor the therapist trainees' skills on 
a variety of dimensions. The Didactic Examination was designed to 
assess therapist trainees* cognitive understanding of tte theoretical 
background, rationale, technigues and procedures of short-term IPT; 
.other instruments were. developed tc assess the. therapist trainees' 
'practical IPT skills in therapy sessions. Besults of analyses of the 
trainee outcome data confirmed the .decision to rely cn lultigle • 
information and derived composite scores. Although, scores on Didactic 
Examinations showed significant improvement at post-testing, the 
written examination had little predictive validity for determining 
the competence of the therapist to perform IPT during practicum. 
Findings of a lack of agreement between supervisor and independent 
evaluator perceptions of the therapist underscore th€ iapcrtance of 
including* videotape reviews of actual therapy sessions as part of 
therapist assessments. t (Ruthor/NBB) 
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•The training of psychotherapists has been an ongoing process 
in psychiatry and clinical psychology. However', Recently ther^ 

has been a growing demand to -operationalize competence criteria 

^ ' * '* 

t to enable independent* evaluation of therapists' skills in 

specifically defined psychotherapies. Despite the incorporation 

of sophisticated design features in many completed psychotherapy** 

efficacy studies, there have been few systematic attempts *to 

explicitly define the essential components of the psychotherapies 

offered in clinical trials. Specification of the treatments 

offeree} is important if different types of. therapy are to be 

compared. Rigorous comparative efficacy studies .cannot be under- 

taken without precise specification of treatment conditions and the 

development of training. and monitoring procedures that are adequate 

to insure that the therapists are performing the treatment as* 

described. In addition to specifying the characteristics of the 

» . * 
therapy being tested, it is essential that procedures for evaluating 

psychotherapy sessions be developed to insure that the therapy being 

practiced is, in fact, the therapy being tested in the clinical 

'trial. 

This paper will describe the evaluation "procedures that were 
developed and 1 tested during a pilot training program in Interpersonal 
Psychotherapy and will examine and compare the utility of 'the various, 
evaluation procedures. We present our experiences as a specialized 
"case repoijf' 1 o£ the development and evaluation of various procedures 
for assessing therapist competence to participate as IPT therapists 
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in controlled, comparative psychotherapy efficacy studies. Before 
describing the evaluation procedures used in the pilot training 
program in Short-Term Interpersonal Psychotherapy (IPT) for 
Depression, some of the defining characteristics of the therapy 
will be outlined. 

IPT Interpersonal Psychotherapy (IPT) is a brief (12r to 16 ^ 

weeks), weekly, individual therap3^TTTc^f^^ on' current social . 

« ■ 

^ancj -interpersonal difficulties in the' ambulatory , non-bipolar, non- 
psychotic depressed patient. The 'therapy, .developed and tested by _ 
the New Kaven-Boston Collaborative' Depression Project (Weissman, 
Prusoff, Pimascio, Neu, Goklaney, & Klerman, 1979; Klerman, DiMascio, 

.Weissman, Prusoff, & Payfcel, 1974) was designed to fill a gap in ■ , 
the field where the only therapies that had been sufficiently 
specified in procedural manuals in order to undertake serious 
replication trials had been those based on cognitive and/or behavioral 
approaches (Beck, Rush, Shaw, Emery, 1980; Klerman, Rounsaville, 
Chevron, Neu, & Weissman, 1979; Luborsky, 1978; McLean, 1980; Rehm & 
Kornblith, 1978). Based' on an interpersonal framework this approach 
utilizes techniques derived from psychodynamically-oriented psycho- 

^ therapies, but treatment- is focused cm .the .patient ' s current life 
jfbd interpersonal relationships. 

When designing procedures for assessing the content^and quality of 9 
the psychotherapy, we gave considerable ^thought^to the following 
methodological issues: What should be ; ,a$sessea? How should it be 
assessed? • . 
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-In terms of what we needed to assess, we recognized two' essential 



types of knowledge that the therapists needed to acquire: (1) conceptual 

V * . 

understanding of the treatment • and .techniques, and (2) the ability 

\\\ * 

to put\this understanding into practice with depressed patients. 



The desired outcome of IPT training was to train therapists 
to understand and perform the tiheYapy as described and avoid 'using 

v« - 

techniques which are not a part .of IPT. Thus, training goals and 
therefore assessment procedures involved evaluation of both 
conceptual skiils and practical skills. Given th^ attempt to make 
IPT representative of -treatments commonly offered for depression, 
many therapist^ accepted into the -training needed to modify their 
style only slightly in order to become effective IPT therapists. 
Thus, the important assessments were not of the extent to which the 
therapist had changed in the course of training but of whether * ^ 
the therapist was utilizing 9 IPT techniques at a level of competence 
Sufficient to certify him/her for participation in a clinical trial. 
With this consideration in mind, change measures were not ' used to 
assess therapist practical skills* \ 

In termiPof the second question, how we assess competence, we 
felt it was important to utilize information from a variety of/ 
sources in order to provide a* relatively complete picture of *V' 
the change process and to contrpi for systematic bias. Researchers 
typically have relied upon information obtained from the therapist, 
'from .the patient, and, in traditional training programs, from .the 
supervisor's global evaluations. .The exclusive reliance on these 
types of ratings is widespread despite the fact that several 
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studi^s have raised questions about the reliability of therapist 
and/or patrent self-reports as inddcators of what actually occurs 
in therapy, . - 

Advances in video technology have made it possible to 4t 



obtain objective recordings of entire therapy hours-^rtiich provide 
a potentially valuable data source for monitoring the process of 
psyctibtherapy. The advantage of utilising videotape recordings 
in psvchotherapy studies is tha t they provide an accurate, more-or-less 
permanent record of the entire therapy which £an greatly facilitate 
the reliable evaluation of the content and quality of therapeutic 
interactions. Thus, in addition to ratings obtained from the 
participants in the therapy pocess, i.e., the therapist, patient 
and supervisor,' we obtained independent documentation of the' 
therapeutic interactions in order to evaluate the reliability of 
various assessment procedures, 1 * 

In conducting the^pilot study to develop 'procedures for . evaluating 
psychotherapist's cb^petence to perform' Interpersonal Psychotherapy 
(IPT), ; a number of <^Ifferent instruments and procedures were 
utilized to monitor the therapist ^trainees' skills on a variety 
of dimensions. By comparing ratings of the therapy process from 



the perspectives of the therapists, the supervisors, ^nd an independent 
evaluator, we demonstrated the importance of obtaining independent 
documentation' of the therapy process. The areas assessed, the 
evaluation instruments/procedures, the raters and timing of the 
assessments are -presented in Table 1. 
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» The didactic examination was designed to assess therapist 
• trainees 1 cognitive understanding, of the theoretical background, 
rationale, techniques and 'procedures of Short-Term Interpersonal 
?sVchotherapv (IPT) . The instrument cpnsisted of 35 questions, in 
'ao multiple choice format, selected on the basis of their relevance 
to the material presented in the didactic seminars; i.e., questions 
that seemed to assess therapists* learning in areas that the 
trainers thought essential to a"" basic understanding of the UT 

V • 

method. The. intent of the trainers in designing the test was not* 

so much to discriminate amongst therapists as to establish that 

* 

each individual ttierapist'had sufficient understanding of the , 
IPT approach to prepare him/her to participate in the practicum 
^portion of . the training program. The* Didactic Examination was 
administered to all therapists before and after ^their participation 
f in the didactic phase of training. 

The overall go^al of a program to train psychotherapists 'for 
efficacy studies is to make sure that they are actuaiiy performing 
theT'tlierapy that is being tested in the clinical trial "and* that they 
are performing it well. Thus, in terms of evaluating the therapist 
trainees' practical IRT skills, the trainers in the pilot program had 

to'defermine what aspects of' the therapists' behavior in treatment 

• * 
should be rated iVorjier to assess whether or not the "therapists were 

actually performing- IPT as described in .the procedural manual. 

The concepts of inclusive and exclusive boundaries were utilized 

in determining the areas to be rated. In order to be performing 

* » 



IPT the therapist must give' evidence of using techniques described in 
the practitioners manual. and follovin| the prescribed general 
strategies »for approaching.pat-ients' problems. On the exclusive N 
side, tfhe therapist should not use technique^ o,r strategies that 
• are definitive of some other -therapy, 

'The instfumeats used to assess the therapist trainee'^ 
practical IPT skills were the Process Rating Form and the* ^ 



Therapist Strategy Rating Fomu The Process -Rat ihg Form wa ( s 
designed to assess therapists' use of the various IPT* techniques 
in each therapy session. Ratings provided information on whether 
or not a particular technique was used, to what extent it was used, 
and on the quality of use. 

The Therapist Strategy Rating Form was used to evaluate therapist 
skill, in'the use of appropriate^ goal-directed activity in relation 
to the interpersonal problem area with which the patient presented. 
IX\ addition to the qualitative rating of the individual therapist's 
use of IPT stragegies, rating were made in the following four areas: 

1. Therapists' skill at helping patifent with intimate self- 

-1 ' ■ 0 \ 

disclosure. \ 

2. Therapists' ability to tend to the therapeutic relationship. 

3. Therapists' ability to focus session on appropriate topic. 



S Overall quality of the session. 

These ratings attempt to*evaluate the overall quality of the 
' psychotherapy, H.e.,' is the therapist generally following good 

clinical pfactice£ ^though* these evaluations assess aspects of 
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the treatment which are ftot specific to IPT, it was felt that if 
trainees were not performing well in these areas they were not . 
performing IPT correctly even if all the" appropriate techniques 

I 

were being used. * * >\ 

^ Using these rating instruments, selected sessions from. the 

beginning, middle, and termination phases of each therapy were ra|ed, 

based on guidelines that were prepared to help standardize the 

*\ 

> qualitative ratings. Prior to the study we established inter-rater 

r * 

reliability among the two trainer/supervisors and the independent 

evaluator, on fhese ratifig forms. 
■ * 

Both of the rating forms ftere* completed by the therapist, 
following each therapy hour, by the independent evaluator on the 
,ba^sis of 'observation of vi4eot^es of entire session^, and l by 
the supervisor, on jthe basis of the material presented by the 
therapist trainee during weekly supervisory Sessions. 

(Global citings '(GR) of the individual ^therapists i^ere completed 

by 3 judges (the supervisor, an IPT trainer, -and an independent 

evalua.tor) on the following dimensions: (1) trainee f s overall 
• • • 

therapeutic skills (pre-existing qualities); (2) trainee T s cognitive 
and practical IPT^ skili^. These ratings wer£ made following 
completion of i\ie training program and reflect, the degree to which- 
the individual therapists wer£ perceived as having demonstrated 
an understanding of IPT concepts and strategies, as reflected in 
the quality of their: - i 
-•^participation in the d id a.cCTcT~t raining seminars;, * 



—discission of case material in weekly supervision and/or^ 

at the weekly case conference; and 
— behavior in therapy, based on the observation of randomly 

selected videotaped therapy segments. 

• * • s 

Evaluations were completed by different raters on the basis of 

different data sources in order to examine and compare the utility of 

the various assessment procedures. In this study ratings by the ^ 

therapist about his own behavior in the therapy hour were used 

v. 

primarily to facilitate learning. The ratings by the supervisor* 

on the basis of supervisory sessions and thej global ratings, were used 

« • 

to tip. the traditional means of determining therapist competence., 

i ^ 
Ratings by an independent evaluator on the basis of videotapes 

of therapy sessions were obtained in .order to provide the most 

objective evaluation of the therapy process and to provide the 

standard by which to-measure" the other ratings.. 

'Because therapist assessment procedures utilized multiple 

outcome measures, gleaned from different perspectives, a' summary 

score was derived in order to reflect a composite estimate of 

each 'therapist's competence. The derived. composite scores reflects 

the overall skill;of the therapist trainee in terms of^jiis/her usre 

of IPT techniques and goal-directed activity. The therapist's 

vselNratings were not included in the imputation of the final 

composite scor€. Althgjagh the therapist trainees- self-ratings 

serve an important function in encouraging the therapists to think, 

* ■ % ■ • 7 ' • ^ u . 

about and assess their work in terfirs ~of -their ^use of IPT techniques ■ 
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and strategies, we do not believe that .these self-ratings should 

Snter into the assessment of therapist competence. Therapist 

V 

self-ratings often reflect therapist modesty more than* they 

reflect an accurate assessment of therapists' skills. •» 

What,, then did we find?? How useful were the various assessm ent 

i 

J procedures? # • 

Results of analyses of the .therapist trainee outcome data 
confirmed our decision to rely on multiple informants and derived 
corcpbsite scores. The Didactic- Examination score,, the outcome 
measure ffor the didactic pha$£ of training, was administered 

prior to the didactic pharse of training as well as after the 

\ • - * ~ 

training was completed. The overall matched. T-test for th^ total 

ifference score (pfe-post) showed significant improvement at 

post-testing (p < .005). However*, the analysis of individual 

items indicated that thG therapists we're familiar with much of v 

the material before training beg;an; half of the litems were scored 

• * 
correctly 0 by all therapists ^t*. that time. 

> ^ j\ , , 

When the therapists post-tr§iiiing scores on the 'written 

♦ • 

examination were correlated with ratings of therapists "practical 4 

skills, there appeared' to be a negative relationship,' although * 

correlations between performance.*on the didactic examination £nd the 

assessments of the therapists' practical IPT%skills failed to 
•S 

reach significance. However, greater improvement on the didactic • 

„ - • t 

examination was. shown tfi.be associated with poorer evaluations 

t 

on the global assessments made by the supervisor/tramers 

• • • \ ' ' .. '- •. J 




(r = .68, p. < .06) and on the Overall Composi^ Score (r = .65, 



p ^ /06). These findings suggest that, at best, there is no » 

* 

relationship between cognitive grasp of I?Y as assessed by the 
didactic examination and competence in^ the practical 'application 
.of IPT and at worst that thes^-skills are irjverselv related . Thus, 
the pilot finding^ have failed to support ^the va^ue of using at 
least this particular written examination as-an assessment of^ ^ 
therapist outcome and may be indicative of the' low general value 
of t^iS kind of assessment in predicting therapjll^s' competence m + 
to perfa^m Dsvchotherapy . w N 

) 

The inter-correlations among -the vario^outcome measures 
used to assess' therapist competence in the pratticum phase of 
training were all in the predicted directfoh; however*, many were 
of low magnitude. The strongest finding which emerged is that 

correlations are most significant when scores derive from the same * 

• ' * * " ./ M 

informant. Thus, the strategy and ^process ratings, completed j 

' — : • - 

by the independent evaluator were highly collated. Similarly, ■ 

the ^supervisor's Strategy Rating Form score correlated significantly* 

with the global assessments, which ^were comprised, in part, f of 

ratings made by the supervisor. That ratings of identical c6ntent 

areas were uncorrelated across cjifferent^ raters may be ^nmtecf ^ 

for, in part, by" the fact that ratinge were^made' from -different 

data sources, i.e., memory, observation of videotape?, and supervision 

Careful initial screening of therapist trainee applicants, which 

tekds to mipimize variance, and -the, samj.1 si^e of the^sampj-'e (n * 9) . 
. » v 1 / 

further reduces the likelihood of obtaining significant inter- 
^correlations. 



\ Although inter-rater reliability on the Therapist Strategy 
Rating Form and .Process Rating Fcwrm was excellent^ when ratings w^re 
made on the basis of videotapes oi\ therapy transcripts, there was 
virtually no agreement between therapist strategy rating? made 
by the supervisors and the strategy or process ratings made by 
the independent evaluator, dne explanation of .these findings 
is that observati^of videotape alldws the rater to view a 
therapy^Sgssiori \n its entirety, whereas, information gained in 
supervision is dependent on a therapist's ability and /or willingness 

to present accurately the therapist/patient interaction in^the 

* * 

therapy hour. 

In summary, an examination of the utility of the various 

therapist evaluation instruments/procedures that were piloted in 
^ $ 

this study indicate that the written didactic examination had very 
little predictive .validity for determining the competence of the 
therapist to actually perform IPT during the practicum phase of 
training, With e respect to tbe practicum phase of training, the •■#^ s * 
evaluation procedures utilized;multiple outcome measures. In most 
clinical training •prgrams, psychotherapists practical skills arfe 
evaluated soleiy, oh the basis of supervisor's global evaluation which 
are based on therapists' reporting of process notes and/or therapists 
presentations at case conferences. Very few programs include < 
detailed ratings of videotapes of psychotherapy sessions as part 
of the assessment procedure, to determine therapists' competence. 
In the present" study, assessments of the trainees were made by 
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several types of raters on the basis of several sources of information, 

including use of traditional supervisor ratings*. 

Although therapistT^setf -ratings serve an important function 

•in encouraging the therapists to think about apd assess their 

^work, in 'terms of their- use of prescribed techniques and Strategics, 

therapists' recollection of sessions and self-rating s ca« be Kiased 

bj self interest. Supervisor's ratings, based on the therapists' 

presentation of process material, share the weaknesses of therapist 

self-ratings, "Ratings of videotapes by an independent evaluator 

9 

who did not have an investment in the process as'either therapist 
or supervisor, and -who had the most complete source of data, appear 
to be the most valid* 

Our findings, which demonstrate' a lack of agreement between 
the supervisor's perceptions' of^the Jtherapist (based on material 
presented in supervision) and the independent evaluator 's perceptions ^ 
' of "the therapist (based on observation of the actual therapy hour), 
raise questions abouf 'trhe customary reliance on supervision as_ the 
sole basis on which to make valid and reliable judgements a£out the , 
* .skill of therapists or 'the technique's and strategies employed in 
the* therapy hour. This finding underscores the importance of 
including'videotape review of actual therapy sessions as part of 
therapist assessment, and, if veplicated in a larger sample, may 
hav* important implications for the selection of £valua£ion procedures' 



*to be utilized in future therapist training programs. 



• ■*> 
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TABLE 1 
ASSESSMENT OF, THERAPISTS 
•Outcome Measures » 



1 

Conceptual Skills 

Understanding of theore- 
tical framework & general 
principles of IPT 

* J 


j 

Didactic Examination 

Ratings 'of therapist's 
participation/ perf6rmance 
in didactic seminars 


Scored exam . 

> 

Trnr./ 
Eval. 


.X' 


X 

X . 


1» 




> 


_ V . 


Practical Skills 

Skili in 'use' of the 
various IPT techniques 
* „ £ 


Process Rating Fornf 4 


v Ind. Eval. 
Therapist 






X 
X 


X 
X 


m X 

X 

a 


X 
X 


' Ability to identify inter- 
personal problem area; 
follow through on goal- • * 
directed activity (focus); 
to help patietft with inti- 
mate self-disclosure; to 
tend to therapeutic rela- 
tionship / 


Therapist Strategy / 
* Ratang From — — 


* 

m 

Supervisor 
Ind. Eval. 
Therapist 






X 
X 
X 


— — n 

X 

X ' 
' X' 


X 
X 

x ; " 


X ~ 

X 

X 


* 

Overall therapeutic skill: 
skill in use of IPT 


* -i r • 
Global Ratings 


v Supervisor" 
Trnr. /Eval #/ 
Ind. Eval. 












X 
X 

X v 


Ove rail - Competence 


; — ^ : 

Derived* Composite Score 














X 



PD** 
*** 



Pre-training 
Pos£-didactic 

At last 'Session or at termination 
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